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ABSTRACT 

Five-hundred d ichotonously scored response patterns 
were generated with sequentially independent (SI) items and 500 with 
dependent (SD) items for each of thirty-six combinations of sampling 
parameters (l.e., throe test lengths, three sample sizes, and four 
item difficulty distributions). KR-20, KR-21, and Split-Half (S-H) 
reliabilities were computed tor these 36,000 response matrices and 
cumulative distribution functions, each based on 500 trials, 
compiled. Comparison of the sampling distributions suggests that 
there is nore similarity between the SI and SD cumulative 
distribution functions for KR-21 than KR-20, and either of these have 
more similarity than S-H. For KE-20, KR-21, and S-H, the similarity 
of sampling d istribu t iojis under SI and SD varies directly with test 
length; the longer the test, the more similar the distributions. On 
the other hand, similarity varies inversely with sample size: the 
larger the sample, the more dissimilar tie distributions. For both SI 
and SD item rets, the expected value of KR-21 is less than or equal 
to that of KR-20 which is less than or equal to the expected value of 
S-H. Also, for both item sets, the standard error of S-H is greater 
than that of KR-21 which is greater than or equal Lo the standard 
error of KR-20. The standard errors ot the three reliability 
estimates vary inversely with sample size and test length, for both 
SI and SD items. In every case, SD increased the expected value of 
the estimate and decreased its standard error. (Authoi/AEj 
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INTRODUCTION 



The general purpose of this study was to investigate the effects of 
sequential dependence (SO) on the sampling distributions of three commonly 
used reliability estimates • Kuder-Richardson Formula ?0 (KR20), Kuder- 
Richardson Formula 21 (KP.21), and odd-even Split-Halves (S-H). 

Two items are sequentially independent (SI) If and only if the 
success or nonsuccess on one does not affect the probability of success 
on the other. Symbolically, if S* represents success on item i and S^ 
indicates failure, items i and j are sequentially independent if and only 
if P(Sj 1 Sj) • P(Si) and P(Si | Sj) • P(Si). 

Two items are sequentially lependent (SD) if and only if the success 
or nonsuccess on one affects the probability of success on the other. 
Symbolically, items i and j are sequentially dependent if and only if 
P(Si I Sj) + P(Si) and P^ | sj) 4 P^). 

Xu practice, it is often the case that achievement tests contain two 
or more items that are SD. The most obvious instance of SD is given by a 
pair of items, one of which refers to the other and the solution of the 
one is strictly contingent upon solution of the other. Frequently, 
however, the sequential dependence between items is more subtle than in 
this extreme case. Since SD does occur in practice and slrce the 
classical reliability estimates are often applied to these Sb tests, it 
la Important to investigate the effects of SD on the sampling distri- 
butions of tM three commonly used estimates. 

In the present study, computational forms of KR20 and KR21 (Kuder 
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and Richardson, 1937) were used, and the S-H coefficients were obtained 
by applying the Speartnau-Brown (Spearman, 1910; Brown, 1910) formula to 
the Pearson correlation between the scores made on the odd and even item 
sets. 

METHODOLOGY 

The following sampling parameters were used: 

Test Length Tests containing 20, 30 and 50 items. 

Sample Size Sample sizes of 20, 32, and 52. 

It em Difficulty Distribution Four different distributions of item diffi- 
culty were used. Three of these distributions were constant difficulties 
of *3, .5, and .7; and the fourth was an equally-spaced rectangular 
distribution of the following form: 

Let i * 1, 2, 3,*«*.,k be the item index over the k items. 

The difficulty for item i is wi- . 

m+T 

In this distribution, the items range in difficulty from — ly to » 

True Score Distributio n For each of the four item difficulty distri- 
butions, a symmetric unimodal distribution of true scores was forced in 
following way: 

Let Di be the difficulty of item i, and n^g, and 
m 3s selected so that: 

D 1 " m ls (*25) D* + m2 S (* 50) + mjg (.25) . 

In this notation, the subscript s is an index over the 
n subjects, and the following relationships hold among 
the weights: 
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m ll c m 12 * ••• ■ n id » where d » n/4 
m 2(d+l) - ra 2(d+2) * B ra 2(3d-l) 
n 3(3d) " m 3(3d+l) * *3* 

These conditions on the weights (ta^, n 2 , m 3 ) give a three-point distri- 
bution of true scores for each parameter combination. The particular 
weights used in the study are presented in Table 1. 

TABLE 1 

WEIGHTS USED TO FORM TRUE SCORE DISTRIBUTION 



Difficulty Distribution 


m ls 


* 2 s 


*3s 


Rectangular 


.5 


1.0 


1.5 


Constant .3 


.5 


1.0 


1.5 


Constant .5 


.5 


1.0 


1.5 


Constant .7 


.7 


1.0 


1.3 



From the preceding discussion the probability of subject r correctly 
answering item i is mi 8 if 1 _< s jc n/4, m 2 s D i if n/4 < s < 3n/4, and 
m 3 a if 3n/4 s n, In this discussion the probability of success on 
an item does not depend on performance on prior items, but only on the 
relative position of the subject's true score. Thus, the items which 
comprise tl % subject's test score are SI. 

SD among items was induced by changing the probability of success on 
a given item according to the subjects performance on the immediately 
Q preceding item. Performance on item 1 is unaffected by SD. 

ERIC 
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In the case of the equally-spaced rectangular distribution of item 
difficulties, P(S^) ■ . The following dependence probabilities were 
used; 



p CS ± I St.!) = 



p(si i s;_!) 



(k-t-l)l-(t 2 -! ) 
(k+1) (k-i+2) 



If D is the constant diffi ulty for each of the three constant item 
difficulty distributions, the dependence probabilities are: 

P(Si I s 1 . 1 ) - J2ktL 

p (St | si-i) «* , 

k-Dk 



It may be easily shown that these dependence probabilities satisfy 
the following necessary condition: 

n r (i) " n -P(Si) - n r (i-1) ‘ P(Si I Sj.i) + 

Ml-l) ’ p < s i I s i-l> » 

where ^ r (i) the number correctly answering item i, n i^ sample size, 
n r(i-l) t * ie num ^ er correctly answering item i-1, and ny^-x) is the 
number incorrectly answering item i-1. 

The procedures of the study required generation of nine sequences of 
pseudo-random numbers in the interval (0, 1). The sequences ranged from 
200,000 to 1,300,000 in length. The multiplicative congruence method 
(Lehaer, 1931) wa 9 employed, using Pike and Hill’s (1965) algorithm, to 
generate the sequences. 

Subject x Item response matrices were generated in an element-wise 
manner by comparing thi probability of success on each item by each 
subject with a pseudo-raudom number. The matrices were generated fn 
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pairs — one matrix containing SI items and the other containing SD items. 
The same random number was used for comparison in each matrix for a 
given position. 

Let XI be the n x k matrix of responses for the SI items and XI) be 
the n x k matrix of responses for the SD items. Also, let XI s i and XD 3 £ 
denote elements of these response matrices. Then, using R g ^ for the 
random number associated with position si, and S s ^ for success on item 
i by subject s, the matrices were generated in the following way: 

If Rgi < P(S si ), then Xl oi - 1 

If R si - P ^ s si)> then XI sl 18 0 

If R s ^ < P(S s £ | then XD^ « 1; otherwise XD g ^ c 0 

If > P(S sl | Sg£_^), then XD S ^ ■ 0; otherwise XD gi = 1 

RESULTS AND CONCLUSIONS 

These Monte Carlo experiments were conducted on an IBM 7094 computer 
using a Fortran Program written by the researcher (copies of the program 
are available upon request)* Five-hundred dichotomously stored response 
patterns were generated with SI items and 500 with SD items for each of 
the thirty-six combinations of sampling parameters (i.e., three test 
lengths, three sample sizes, and four item difficulty distributions). 

KR20, KR21, and S-H were computed for each of these 36,000 response 
matrices and their cumulative distribution functions, each based on 500 
trials, compiled. 

The Kolmogorov-Smirnov two-sample procedure (Siegel, 1956) was used 
to compare the sampling distributions of the three estimates under SI and 
SD. Kolmogorov-Smirnov D values were obtained by computing the absolute 
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value of the maximum vertical distance between the cumulative distri- 
bution functions. Considering all parameter combinations, the mean 
absolute D values for the three estimates were: 

KR20 KR21 S-H 

.307 .303 .333 

Since Kolmogorov-Smirnov D values vary inversely with distribution 
similarity, it may be concluded that there is more similarity between the 
SI and SD cumulative distribution functions for KR21 than KR20, and 
either of these have more similarity than S-H. 

Mean absolute D values were also computed for the SI and SD 
cumulative distribution functions under the various sampling conditions. 
These means are presented in Table 2. 

For KR20, KR21, and S-H, the similarity of their sampling distri- 
butions under SI and SD varies directly with test length, i.e., the 
longer the test, the more similar are the sampling distributions. On the 
other hand, the similarity of the sampling distributions of the three 
estimates under SI and SD varies inversely with sample size, i.e., the 
larger the sample size, the more dissimilar are the sampling distributions. 
The different item difficulty distributions effect different degrees of 
similarity between SI and SD distributions. For both KR20 and KR21, the 
order of similarity of the sampling distributions under SI and SD is: 
constant .3, equally-spaced rectangular, constant .5, and constant .7^ 

For S-H, the order is: equally-spaced rectangular, constant .3, constant 

•5, and constant ,7. 

For both SI and SD item sets, the expected value of KR21 is less than 
or equal to that of KR20 which is less than or equal to the expected value 
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TABLE 2 

MEAN KOLMOGOROV- SMIRNOV ABSOLUTE D VALUES 
UNDER THE VARIOUS MONTE CARLO CONDITIONS 



Difficulty Distribution 


XR20 


KR21 


S-H 


Rectangular 


.238 


.229 


.274 


Constant ,5 


.385 


.350 


.378 


Constant .3 


.196 


.204 


. 285 


Constant .7 


.403 


.427 


.397 


Test Length 


KR20 


KR21 


S-H 


20 


.422 


.427 


.463 


30 


.309 


.290 


.333 


50 


. 190 


.185 


.205 


Sample Size 


KRcO 


KR21 


S-H 


20 


.261 


.265 


.274 


32 


.297 


.303 


.329 


52 


. 364 


.339 


.397 



of S-H. Also* fcr both SI and SD item cels, the standard error of S-H is 
greater than that of KR21 which is greater than or equal to the standard 
error of KR20. The standard error of each of the three reliability 
estimates varies inversely with sample size and test length, for both SI 
and SD items. In every case, SD increased the expected value of the 
estimate and decreased its standard error. 
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